Field of the Invention
The present invention relates to an image processing apparatus and, more particularly, to an image processing apparatus for processing a picked-up image obtained by an image pickup element having a microlens array at a front side thereof.
Description of the Related Art
In the related art, there exist many image pickup apparatuses such as an electronic camera and the like for recording and reproducing a still image or a moving image picked up by an image pickup element such as CCD, CMOS, or the like to/from a recording medium serving as a memory card having a memory element.
An example of techniques regarding those image pickup apparatuses has been proposed in Japanese Patent Application Laid-Open No. S61-261974. Such an image pickup apparatus is constructed in such a manner that a microlens array (hereinbelow, abbreviated to an MLA) in which microlenses are arranged at a rate of one to a predetermined number of plurality of pixels is arranged at a front side of an image pickup element, thereby enabling incident direction information of a ray of light (ray directional information) which enters the image pickup element to be also obtained.
Although such an image pickup apparatus is called a Hand-held Plenoptic Camera in Ren. Ng. and 7 others, “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02, it is generally also called a “Light Field Camera”.
As an application of such an image pickup apparatus, a normal photographed image is generated on the basis of an output signal from each pixel. As another application, a predetermined image processing based on the ray directional information is executed to a photographed image, thereby reconstructing an image focused to an arbitrary focal length (such a processing is called a refocus processing), or the like.
On the other hand, among recent electronic cameras, there exist many cameras each of which is equipped with an image pickup element having millions to tens of millions pixels. However, it is practically very difficult to manufacture an image pickup element in which all pixels properly perform a conversion into an electric signal corresponding to an incident light amount. Thus, several “defect pixels” which do not normally operate exist mixedly in the “pixels” of the image pickup element.
In the image pickup apparatus in the related art, an interpolation processing or the like using an image signal of a peripheral pixel of the defect pixel is executed, thereby correcting an image which is finally generated.
There are several methods of detecting the defect pixel which is a correction subject. For example, in a manufacturing step of the image pickup apparatus or image pickup element, methods which record and store an address of the defect pixel in a memory of the image pickup apparatus on the basis of the photographed image have been proposed in Japanese Patent Application Laid-Open No. S61-261974, and Japanese Patent Application Laid-Open No. 2005-026794, and the like.
As another detecting method, there is a real-time defect pixel detecting method of discriminating the defect pixel based on a level difference or the like from the image signal of the peripheral pixel of the picked-up image every photographing by the image pickup apparatus. Such a method has been proposed in Japanese Patent Application Laid-Open No. 2005-286825, or the like.
However, in the photographing by the image pickup apparatus having the MLA as mentioned above, in order to realize the real-time detection of the defect pixel, there is the following problem.
That is, in the real-time detecting method of the defect pixel, on the basis of the level difference between the detection subject pixel and its peripheral pixel, whether or not that detection subject pixel is a defect pixel is discriminated. However, among the image pickup apparatuses having the MLA mentioned above, the image pickup apparatus based on a prerequisite that the reconstruction for executing the predetermined image processing such as a refocus processing or the like is performed has the following problem. That is, an arrangement of the pixel signals at the time of reading out from the image pickup element or at the time of storing the image data is not always an arrangement corresponding to a positional relationship of the object images, and thus the image data is of data which may not be suitable for direct display to observe. Therefore, if such an arrangement is used as it is, it is difficult to discriminate whether or not the detection subject pixel is a defect, based on the peripheral pixel signal of the defect pixel as shown in Japanese Patent Application Laid-Open No. 2005-286825.
But, in order to detect the defect pixel, if some reconstruction processing is executed in order to obtain the image data which is suitable for direct display to observe the image which is used for detection, one pixel signal of the image obtained after the reconstruction becomes a signal generated by adding pixel signals of the plurality of image pickup element. Consequently, it is further difficult to detect which pixel of the image pickup element is a defect pixel, based on the pixel signal of the reconstruction image.